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COLOR CORRECTION OF MULTIPLE COLORS 
USING A CALIBRATED TECHNIQUE 



Background 



A typical color camera separately monitors red, green, and 
blue components of an image. Electronic imagers rely on some 
means of separation of the illumination incident on the sensor 
into a number of spectral channels. Widely-used means of 
discrimination between spectral components of the imaging scene 
include the Color Filter Arrays (usually employed in systems 
based on single focal plane array) and beam-splitting prisms 
(primarily employed in systems based on multiple focal plane 
arrays) . In either case, the incident illumination is separated 
into a small number, usually 3 or 4 , discrete color (spectral) 
channels . 

For any given scene, the exact ratio of color channel values 
will depend on the spectral characteristics of the entire optical 
stage of the imager. This is typically done using a color filter 
array over a photosensitive area. The color filter array is 
formed of a plurality of different colored elements, which 
respectively pass only color of a predetermined spectral 
parameter . 

A typical color filter array is shown in FIG. 1A. Each of 
the boxes such as 100 represents a single pixel. Each set of 
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four boxes outlined by the line 102 can be considered as a 
megapixel. The pattern in the megapixel repeats throughout the 
entire color filter array grid. 

Spectral channels of the modern color imagers do not have 
the same spectral sensitivities as color-sensitive elements, the 
cones, of the human eye. FIGs. IB - ID respectively illustrate 
the spectral sensitivity curves of the human eye (ft,y,p) and 
spectral transmittances of the red, green and blue color channels 
typical for modern RGB electronic imaging systems. As a 
consequence, the direct use of individual color channels values 
of the imager as stimuli for typical display devices, such as VGA 
or NTSC monitors, does not lead to correct color rendition. 
Hence, colors are corrected color signals detected by the 
electronic imaging system into color channel stimuli appropriate 
for the output to the image rendering device . 

Each megapixel 102 has two green filters, one red filter, 
and one blue filter. This is because the eye is usually more 
sensitive to green than it is to red and blue. 

The intent of the filter of FIG. 1A is to provide an image 
which is precisely matched to the spectral content of the eye. 
However, this filter, while it does the best that it can, is not 
precisely matched. It is often desirable to interpolate between 
the values. 
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For example, the value received at area 100 is only 
indicative of the green portion impinging on area 100. However, 
some part of that incoming light is also red. Another part of 
the incoming light is also blue. Hence, each of the pixels is 
5 processed according to a transformation to solve the equation: 



R = K 1X R + K 12 G + K 13 B 



G = K 21 R + K 22 G + K 23 B 
B = K 31 R + R 22 G + K33B 



%p This transformation completely defines the system. This 

CO 

l5>§ includes the so-called color transformation matrix: 
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15 Summary 

The inventor recognized from the above, however, that this 
corrects only for red, green, and blue parts of the image. The 
actual red, green, and blue colors look good. The other colors 
are not corrected quite as well. 
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The present system corrects the signal for all of a 
plurality of desired colors. This is complicated, however, by 
the fact that one only has control over red, green, and blue. 

According to the system as disclosed herein, the operation 
is broken down into a number of colors, e.g., 24 colors. The 
camera is used to detect a color chart or a chromaticity chart 
which has the 24 colors of interest. Each detected color is 
compared with a reference color. The coefficients of the color 



*U coefficient matrix are adjusted to make each of those colors 

iU 

lffip approach the reference color more exactly. Another aspect 



weights the colors so that the most important colors obtain more 
correction . 
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^ ^ w^r^^^a- Brief Description of the Drawings 

1% E These and other aspects of the invention will be described 

in detail with respect to the accompanying drawings, wherein: 

FIG. 1 shows a prior art megapixel system used a color 
filter array; 

FIGs. IB- ID illustrate the sensitivity of the eye to 
20 different colors; 

FIG. 2 shows a basic setup of the system of the present 
embodiment ; and 

FIG. 3 illustrates a typical calibration chart. 
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Description of the Preferred Embodiments 



The conventional techniques to determine the coefficients 
Kij of the color correction matrix are based on analyzing the 
image of color calibration chart with N distinct color elements. 
In order to calculate a color correction matrix from the image of 
the color calibration chart, the appropriate stimuli Yi, suitable 
for driving a given image rendering device, must be known for 
each element of the chart from an independent experiment . Such 
color measurement techniques are, for example, specified by the 
Commission on Illumination (CIE) . Each of N distinct elements of 
the calibration color chart Yi are known. Xi are determined 
based on imaging of the chart. The coefficients of the color 
correction matrix solve N 2 linear equations or solve N equations 
for each chart element . 

Color correction matrix obtained by this technique provides 
for correction of color rendition of all N colors used in the 
process, provided that the colors could be physically displayed 
by the image rendering device and that they are within the 
spectral sensitivity range of the imaging system. 

Even though the color correction matrix obtained by this 
approach provides for good rendition of the N colors used in 
calibration, other target colors are not rendered correctly. 

In fact, in many real imaging systems once the basic colors 
(such as red, green and blue in RGB systems) have been 
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calibrated, the white elements of the scene appear as pink or 
blue. To counteract this effect, many electronic imaging systems 
employ white balancing techniques subsequently to color 
correction. White balancing algorithms aim at achieving good 
rendition of white color by adjusting (dynamically or statically) 
the gains on individual color channels. This procedure, in 
effect, modifies the coefficients of the color correction matrix. 
However, modification of the color correction matrix inevitably 
leads to deterioration of the rendition quality of the basic 
colors used in color calibration. For example, in RGB systems, 
the good quality of rendition of white is achieved at the expense 
of somewhat arbitrary degradation of red, green and blue. In 
extreme cases, white balancing may lead to the situation where 
only white color is rendered correctly and most other colors are 
visibly distorted. 

In general, it is difficult to achieve "perfect" color 
rendition across the entire visible spectrum using electronic 
imaging system with finite number of color channels. Since the 
spectral sensitivities of realistic imaging systems and the human 
eyes do differ at every wavelength, every color representable by 
the image rendering device would have to be corrected 
individually. For example, a VGA monitor with 24 bits per pixel 
(8 bits per color channel) would require color correction matrix 
with 2 24 rows to achieve the best possible color rendition. 
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This system selects a number of colors including at least 
the primary colors (3) , white, and at least 1-20 other colors, 
preferably at least 3 others. This generates the NxN color 
correction matrix for any given N-channel imaging system that 
would yield the best possible compromise in quality of the 
rendition of the large numbers of colors, as opposed to 
perfecting rendition of N colors at the expense of others. All 
of these colors must be balanced. 

The preferred setup is shown in FIG. 2. A color chart 200 
has thereon a plurality of desired colors. While the preferred 
embodiment uses 24 colors, any number could be alternately used. 
Any number of colors, greater than the three primary colors (red, 
green, and blue or cyan, magenta, yellow) could be used. A video 
camera 202 obtains an image of the color chart 200, which is 
processed in the image processor 204. Preferably those other 
colors include at least white, and at least three other preferred 
colors, but more preferably the other colors include 20 other 
colors total for a total of 24 colors. 

Image processor 204 compares each detected color with a 
reference color. 

The color calibration chart has a number of distinct color 
elements corresponding to the colors that are deemed important 
for overall good color rendition of the electronic imager. For 
an N-channel imager, the proposed technique does not impose any 
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limitation on the number of colors to be optimized, as long as 
the calibration table contains at least N+l distinct color 
elements. The color channel stimuli Yi appropriate for the given 
image rendering device (such as RGB values for VGA monitor) 
should also be known for each color element of the calibration 
chart from an independent experiment . 

The techniques for characterizing the calibration chart are 
beyond the scope of the present disclosure. There is a number of 
commercially-available and well known color calibration charts 
characterized by their manufacturers in accordance with standard 
techniques set by standard-defining organizations such as Inter- 
Society Color Council at National Bureau of Standards (ISCC) and 
International Commission on Illumination (CIE) . 

jJj^T^^ji order to generate an optimal color correction matrix, the 
Calibration chart is imaged under illumination conditions similar 
to those Vhat will be used during normal operation of the imager. 
Once the image of the chart is obtained, the detected signal 
values of each color channel Xi of the imager are recorded for 
all color elements of the calibration chart. 

For each of the colors to be corrected, three least-squared 
fits are carried out. The least-squared fits are done for each 
of the primary colors; here, red, green, and blue. An error 
signal G E is obtained by taking the square of the difference 
between Gn' (what one expects to see for the green color) 
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subtracted from the G actual (what one actually sees) . Similar 
operations are done for red and blue. 

The mathematical representation of this operation is shown 
in Equation 3 . 

5 (Gn' [what expect to see] - G c [actual] ) 2 = G E 

(Rn' - R c ) 2 = R E ... (3) 

a (Bn' - B c ) 2 = B E 

*f" The goal of this technique is to minimize G E , R E/ and B E for 

O all of the 24 colors, all for all of the desired colors for which 

P3 

oHJ correction is to be carried out. The coefficients in the color 

correction matrix of Equation 2 are adjusted in order to minimize 
p the 24, three-item sets. This can be done by trial and error, or 
by solution using linear equations to dynamically change all the 
values until the best mix is reached. The minimization can be 
15 expressed mathematically as 
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min 



y=l L\ ,=i 1 / 



where K is the number of the color channel currently being 
calibrated, M is the number of color chart elements used for 
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calibration, and N is the number of the color channels in the 
imaging system. 

Hence, this system adjusts all the values until making the 
best compromise. 

A second embodiment notes that color calibration based on 
Equation (2) results in the color correction matrix that yields 
comparable quality of color rendition for all colors in the 
calibration chart. However, the distortion of some colors 
impacts subjective image quality more than the distortion of 
other "less important" colors. For example, poor color rendition 
of white, gray and skin colors is more noticeable than poor 
rendition of blue and yellow parts of spectra. 

This embodiment, therefore, modifies the color calibration 
process to attach more importance to some colors at the expense 
of the others. In the context of proposed least -squares based 
method, the ability to prioritize colors is achieved by using 
weighting coefficients multiplying individual color deviations. 

Therefore, in this embodiment, the color correction matrix 
is evaluated based on minimization of the weighted sum of squares 
of differences between detected color channel values Xi and 
corresponding stimuli Yi across entire set of color elements of 
the calibration chart as follows: 
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C C C 3 J=l LV /=1 J J 



where Wj is the weight given to the color of the j-th element of 
the calibration color chart. 

s uses the equations shown as Equation 4. 



\ (Gn' - G c ) 2 • W ± = G E 

(Rn' - R c ) 2 • Wi = R E ... (4) 
(Bn' - B c ) 2 • W S = B G 

Where J is between\l and 24 and each of the J's representing a 
different color, thrs second embodiment adds a weighting factor 
Vl i . Each color is weighted according to its importance in seeing 
an image the way the ew expects to see the image. Practically 
speaking, red, green, anavblue are extremely important, white is 
important, skin color and gray scale are also important. 
Unimportant ones are brown ar^ other dull colors. All of the 
rest are medium importance 
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Since this system takes care of all colors simultaneously, 
one obtains the best possible trade-off between all of the 
different possibilities. 

preferred mode uses a 3 -color RGB Byer pattern color 
filteA assembly . The system uses 24 -color GretagMacbeth 
ColorCh^cker chart marketed by GretagMacbeth, New Windsor, NY as 
the coIok calibration chart. GretagMacbeth describes their chart 
_ as "Checkerboard array of 24 scientifically prepared color 
€1 squares in\a wide range of colors. Many of these squares 
lffi represent natural objects of special interest, such as human 
%o skin, foliaga and blue sky. These squares are not only the same 
fy color as theiz\ counterparts, but also reflect light the same way 
Li in all parts of\ the visible spectrum. Because of this unique 



feature, the sqiiares will match the colors of natural objects 
rpr under any illumination and with any color reproduction process." 
^ This preferred mode assigned higher weights to chart 

elements corresponding to red, green, blue and human skin 
elements as well as a number of greyscale elements. 
FIG. 3 shows an exemplary ColorChecker chart. 
20 Another aspect of the present specification is the approach 

to white balance procedures in conjunction with use of the color 
correction matrix generated by described above method. As 
mentioned in the Background, the conventional approach to white 
balancing is to adjust the pre-computed color correction matrix 
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so that the image areas corresponding to white colors appear 
white. This is usually done by adjusting individual gains on the 
color channels of the imager to achieve equality of all color 
components for white areas of the image. However, the byproduct 
of this procedure is that the quality of the color rendition for 
colors used during calibration of the color correction matrix is 
being somewhat arbitrarily compromised. In contrast, the color 
correction method disclosed here results in optimal balance 
between quality of rendition of white as well as other important 
colors that were given high weights during color calibration. 
Thus optimal white balance is "built-in" the color correction 
matrix and does not require further adjustments as long as the 
spectra of the illumination does not change. 

Another important reason for using dynamic real-time white 
balancing in imaging systems is to compensate for possible 
changes in the illumination spectra. From this point of view, 
dynamic white balancing will be beneficial to systems calibrated 
according to the present invention as well. However, since the 
initial color correction matrix determined from Equation (3) 
already provides white balance under the same illumination 
spectra as was used for calibration, the traditional methods of 
dynamic white balancing should be modified. 

In systems calibrated in accordance with the present system, 
the aim of the dynamic white balancing is to keep ratios of the 

-13- 



PATENT 

ATTORNEY DOCKET NO: 08305/050001/98-26 



color channels of the imager at white areas of the image equal to 
the ratios of the same color channels measured for white image 
areas in the same illumination conditions as existed during 
calibration. This approach allows to dynamically compensate for 
changing illumination spectra without favoring the quality of 
rendition of white at the expense of other colors. 

Although only a few embodiments have been described in 
detail above, those of ordinary skill in the art will understand 
that modifications are possible without departing from the 
teaching noted above. 
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